A Comparison of Reinforcement Learning Methods for Automatic Guided Vehicle Scheduling

Automatic Guided Vehicles or AGVs are increasingly being used in manufacturing plants for transportation tasks. Optimal scheduling of AGVs is a difficult problem. A learning AGV is very attractive in a manufacturing plant since it is hard to manually optimize the scheduling algorithm to each new situation. In this paper we compare four reinforcement learning methods for scheduling AGVs. Q-learning[Watkins and Dayan 921 and R-learning[Schwartz 931 do not use action models. Q-learning optimizes the discounted total reward, while R-learning optimizes the average undiscounted reward per step. ARTDP[Barto et al. to appear] is a discounted method that uses action models. H-learning[Tadepalli and Ok 941 is an undiscounted version of ARTDP based on an algorithm of Jalali and Ferguson[Jalali and Ferguson 891. In our domain(see Figure l), there are two queues generating jobs, an AGV, a moving obstacle and two lanes. Queue 1 generates jobs for lane 2 half the time and Queue 2 always generates lane 1 jobs. The task of AGV is to move jobs from the queues to their destination lanes while avoiding collisions with the obstacle, which randomly moves up and down. There are a total of 540 states. At any time an AGV may do nothing, load, unload, or move up, down, left or right. The goal is to maximize the average reward per step. An experiment compared Q-learning, R-learning, ARTDP and H-learning in our AGV domain. The reward is -5 when the AGV collides with the obstacle, +5 when it unloads a job to lane 1, and +l when it unloads a job to lane 2. Figure 1 shows the medians of average reward per step over 30 trials evaluated separately after turning off learning at various stages. To enable exploration, 50% of the time a randomly chosen action was excuted during learning. The parameters of ARTDP, Q-learning, and R-learning were tuned to this domain by trial and error. ARTDP with discount factor y=O.9 and Q-learning could not converge to the optimal policy even after 2 million steps. Even though R-learning and ARTDP with high y converged to the